perm filename TEN.1[DOC,TOB] blob
sn#050511 filedate 1973-06-19 generic text, type T, neo UTF8
00100 TEN YEAR PROJECT REPORT
00200
00300 VISION HARDWARE
00400 flying spot
00500 tv camera interface
00600 image dissector
00700 instrument tv, pan/tilt/color/focus/turret/target
00800 laser
00900 second camera
01000
01100 VISION SYSTEMS
01200 edge follower
01300 edge operator
01400 accomodation
01500 calibration
01600 simple/complex
01700 visual feedback, corner finding Wichman,Gill,Davis,Perkins
01800 motion/differencing
01900 motion stereo
02000 representation
02100 displays and geomed
02200 description of curved objects
02300 outdoor scenes texture, Ruzena and Yoram
02400 Grape
02500
02600 ARM HARDWARE
02700 Rancho arm
02800 controller, pulse width
02900 hydraulic arm
03000 Scheinman arm
03100 tachs
03200 tools
03300 touch and force
03400 two hands
03500 turntable
03600
03700 ARM SYSTEMS
03800 computer servo
03900 Pieper-kinematics and collision avoidance
04000 Kahn-dynamics, bang-bang servo
04100 block manipulation (Paul)
04200 hand language-move,wave
04300 Newtonain servo dynamics and trajectory calculation
04400 adaptive control with force and touch
04500 two hands
04600 assembly task
04700 high performance arm control (pre-computing trajectories)
04800
04900 SYSTEM SOFTWARE
05000 hand/eye system
05100 SAIL
05200 hand/eye library
05300
05400
05500
05600 DEMONSTRATIONS
05700 water pump
05800 instant insanity
05900 screwing and cranking
06000 putting blocks in holes
06100
06200
06300 Breaking arm control into servo and planning elements,
06400 and by precomputing trajectories, it is possible to have
06500 intelligent choice of trajectories and still have a small
06600 system which runs on a minicomputer-the mini and maxi systems.
00100 The project has progressed to several solid accomplishments:
00200 manipulation capabilities which have enabled a demonstration of
00300 automated assembly of a water pump; a representation for the shape of
00400 complex objects, laser ranging apparatus, and programs which form
00500 symbolic descriptions of complex objects such as a doll or toy horse;
00600 initial achievement in labelling sky, road and trees in outdoor
00700 scenes, using a world model.
00800
00900 We have now focussed a major effort on robotics in industrial
01000 automation. That focus unites work on strategies for assembly,
01100 perceptual strategies, representation of shape and modelling of
01200 objects, visual feedback and manipulation. We are now working on a
01300 sequence of sharply defined tasks and abilities, and are constructing
01400 a system which not only incorporates these abilities, but requires
01500 them to work together.
01600
01700 Recent progress has depended on previous system building. The
01800 achievement of one year in manipulation research followed three years
01900 of design and construction of hardware, and programming adequate
02000 basic arm control and planning software. Even that effort depended
02100 upon the experience of programming the Rancho de Los Amigos
02200 prosthetic arm, and building a hydraulic arm.
02300
02400 RESEARCH IN MANIPULATION
02500
02600 The group began by acquiring and interfacing a prosthetic arm from
02700 Rancho de los Amigos hospital. Although the device was mechanically
02800 inadequate, even after considerable attempts at improvements, the
02900 software experience was valuable. Computer servo was used from the
03000 beginning. Computer control was to prove much more versatile for
03100 control of the new arm. A program was written to provide point to
03200 point control for arm movements. In order to move through crowded
03300 workspaces, a program was written to avoid obstacles while carrying
03400 out arm movements []. demonstrations of these capabilities were
03500 performed [], but a great deal of unnecessary work went into making
03600 up for deficiencies of the hardware.
03700
03800 A hydraulic arm was designed and built according to design criteria
03900 of speed and strength. Arm control software was converted to this
04000 arm and another type of servo was developed, minimal time servo, ie
04100 bang-bang servo. The effect was impressive but not useful. The arm
04200 was found unwieldy for software development; that experience made
04300 clear some design criteria for a new arm.
04400
04500 The next arm was designed with software in mind []. The arm was
04600 completed in December 1970, and has proved a very good research
04700 manipulator. Several other groups have made copies of it. A smaller
04800 version of the arm is under construction and promises to become a
04900 laboratory standard, particularly since its cost is approximately
05000 $3500 each in quantities of two.
05100
05200 Arm control software was developed concurrently, and a few months
05300 after the hardware was operating, the software was operational. The
05400 software was split into a small arm control servo program which ran
05500 in real time mode on our PDP-6 computer, and a trajectory planning
05600 program written in a higher level language, SAIL. This split has
05700 proved to be very useful, in ways which we discuss later.
05800
05900 The arm servo software contained several new features: a) feedforward
06000 from a Newtonian dynamic model of the arm, including gravity and
06100 inertial forces; b) critically damped harmonic oscillator feedback
06200 including velocity information from tachs; c ) trajectory
06300 modification facility in the servo program, which allows considerable
06400 changes from planned trajectories to accomodate contingency
06500 conditions.
06600
06700 Compare this control mechanism with the usual analog servo; the
06800 kinematic model and computer servo allow changes of several orders of
06900 magnitude in the equivalent servo constants, to account for
07000 variations in gravity loads and inertial effects.
07100
07200 The arm was designed so that solution for joint angles from
07300 position/orientation is simple and rapid. Its geometry is the same
07400 as the Unimate []. With a change of solution, the techniques apply
07500 to a wide range of arms; thus our work has significance for
07600 industrial and other robotics applications.
07700
07800 The MOVE_INSTANCE capability shows a simple form of automating some
07900 manipulation procedures. The routine chooses a best grasping and
08000 departure for a class of known objects. Models of these objects are
08100 stored in order to choose all possible grasping positions-parallel
08200 faces, a face and a parallel edge, etc. The full range of possible
08300 departure and approach angles is investigated and a solution chosen,
08400 if possible, which allows manipulation in a single motion. Otherwise,
08500 a solution is chosen which uses an intermediate position to regrasp
08600 the object.
08700
08800 As yet, no new forms of obstacle avoidance have been programmed, so
08900 that the only facilities are the simple avoidance of collision with
09000 the table and the arm post, and the very local technique of Pieper
09100 which is slow[].
09200
09300
09400 Arm planning is based on making complete motions, eg picking up an
09500 object and putting it down. If we pick up an object arbitrarily,
09600 without thought for putting it down later, we may not be able to put
09700 it down as desired and may need to grasp it again. Because arm
09800 planning is separated from arm servo control, pre-computing of
09900 trajectories allows high performance arm motions in repetitive
10000 situations.
10100
10200 Complete trajectories are pieced together from various pieces, eg
10300 grasp, departure, mid-segment, approach, release. Trajectories are
10400 specified by the user in an interpretive hand language (HAL) in terms
10500 of macros at the level of inserting a screw. Endpoint positions and
10600 orientations are often specified by positioning the arm itself in the
10700 proper position and recording joint angles, a form of "learning by
10800 doing".
10900
11000 The project is unique in the level of accomplishment in computer
11100 controlled autonomous manipulation. Still other advances are close
11200 at hand. A second arm is about to be operational, and we will be
11300 developing two hand cooperation. For industrial applications this
11400 seems essential, since many assembly operations could be done by a
11500 blind man, but not with one hand.
11600
11700 INDUSTRIAL APPLICATIONS
11800
11900
12000
12100 There are many reasons for believing that the level of manipulation
12200 which we have demonstrated, and some simple vision capability, have
12300 important industrial applications. There are a class of tasks which
12400 can only be done by machines, not humans, eg reactor assembly and
12500 maintenance. In some cases, availability not cost is crucial, and
12600 some intelligence is has great value in interfacing to humans.
12700 Industrial robots already have some market (a few million dollars per
12800 year) and we can foresee that upgrading them with sensors which add
12900 very little to cost can have great impact on their utility. Our own
13000 designs give us some feeling for costs so that we can see that for
13100 small scale assembly, system costs of $10k for two arms, touch and
13200 force sensors, and a minicomputer. The falling cost of computation
13300 is an important factor. Finally, the benefits for product quality
13400 and design techniques are considerable. We have made some of these
13500 arguments in detail in a proposal for support of our industrial
13600 automation effort [] and they will be only briefly summarized below.
13700
13800 As a demonstration of assembly technique, we have combined vision and
13900 manipulation in an assembly of an automobile water pump. The
14000 assembly used a computer controlled tool, a powered hex-head
14100 screwdriver. A film shows the process quite adequately. We will go
14200 on to assembly of a much more complicated device, which will further
14300 aid in development of our techniques.
14400
14500 We think that the robotics capabilities and the cost factors
14600 constitute a "technology push" which will move designers to
14700 incorporate such devices. Rough cost factors for such a general
14800 assembly device in quantities of 1000 are: two small Scheinman arms
14900 $1000 each including hand, touch and force sensing $800, arm drivers
15000 and interface $1000, small computer with 8k memory $4500, total
15100 $8300. We assume that the price is twice the cost, or about $20,000.
15200 Unimation now projects labor costs at $22000-$24000 per year averaged
15300 over the next ten years, and assumes a payoff over 2 years for an
15400 acceptable cost of $46,000.
15500
15600 Two conclusions appear from industrial representatives at various
15700 workshops: automation should be programmable; automation should be
15800 incrementatl. We contend that incremental incorporation of
15900 programmable assembly devices requires a sophistication of the
16000 assembly device which requires further directed research, but that it
16100 is within reach.
16200
16300 The improved quality of automated processes cuts costs of repair and
16400 warranty, and decreases inspection costs. There are important design
16500 advantages in programmable automation, since much smaller design
16600 costs are required for small changes or for small lot production. In
16700 some cases it is a question of automating processes which are not
16800 otherwise automatable; in others, it is a case of advantages over
16900 "hard" automation. Briefly summarized, some advantages are: economies
17000 of scale in hardware quality and cost; economies of intellectual
17100 scale (commonality of design); longer life for tools; more
17200 flexibility in design and modification; more efficient scheduling and
17300 better utilization; less capital tied up in stockpiling.
17400
17500 A primary research effort is required for programming assembly
17600 robots. Although robots will be useful even if programming is
17700 moderately painful, their full impact will come when changing jobs is
17800 simple. We are planning to attack that problem by building an
17900 interactive system combining an APT-like language, graphics
18000 techniques for displaying 3-d objects, with vision and manipulation
18100 primitives. "Learning by doing" is already available, and we will
18200 combine that with the symbolic abilities just indicated. An
18300 attractive possibility is that of providing the designer with a
18400 simulation system which will aid "design for automation" by giving
18500 feedback on what is achievable by the programmable assembly devices.
18600 Simultaneously, a program for the assembly device would come from the
18700 design process.
18800
18900 We have conceived of an assembly module as a system controlled by a
19000 small configuration minicomputer, which could stand alone. The
19100 programming of the module would best be done on a system with
19200 graphics problem-solving, and large memory, either a central
19300 time-sharing system or a remote system. There is great advantage in
19400 combining assembly modules with "computerization of the factory".
19500
19600
00100 VISION RESEARCH
00200
00300
00400
00500 During the past two years our vision effort has shifted from scenes
00600 of blocks to outdoor scenes and scenes of complex objects such as
00700 dolls or toy horses. In both cases, interpretation has made use of
00800 world models.
00900
01000 A crucial part of our work with complex objects is our development of
01100 suitable representation of shape. This is an area closely connected
01200 with industrial applications of research, since representation of
01300 shape is important in programming of robots, design, display, visual
01400 feedback, assembly, and symbolic description for recognition. A
01500 class of representations of shape has been evolved [], and applied to
01600 symbolic description of shapes of toys and tools []. The
01700 representation depends on a part/whole description of objects in
01800 terms of parts which may themselves be composed of parts. The
01900 success of any part/whole representation depends on the utility of
02000 the primitive parts; within the representation, dominant primitives
02100 are "generalized cones", originating from a formalization called
02200 "generalized translational invariance" []. These primitives are
02300 locally defined by an arbitrary cross section and a space curve
02400 called the axis, along which the cross sections are translated normal
02500 to the axis.
02600
02700 Local pieces of primitives are strung together continuously to
02800 generate complete primitives like a screw, doughnut, snake, etc. A
02900 complete primitive is a cross section which varies continuously along
03000 a continuous space curve called the axis, always normal to the axis.
03100 A rich set of primitive shapes can be generated from simple local
03200 models, and the primitives can be combined in a wide variety of ways.
03300 Complete descriptions are graphs of structured parts. The local
03400 models provide an appropriate level of semantic guide to control
03500 local description processes.
03600
03700 We have developed laser depth ranging hardware which has been
03800 operative since January 1971. The device is very simple and can be
03900 replicated now for less than $1000, assuming that a suitably
04000 sensitive camera tube such as a silicon vidicon or solid state sensor
04100 is available. From that experimental data, we have obtained complete
04200 descriptions of a doll, a toy horse, a glove, etc, in terms of
04300 part/whole descriptions in the representation just described. The
04400 same techniques could be used for monocular images, with considerable
04500 benefit, in spite of the added difficulty of using only monocular
04600 information. We are now writing programs which match these graph
04700 descriptions for recognition.
04800
04900 The representation is also useful for work in visual feedback. It is
05000 relatively simple to describe many shapes we are interested in, and
05100 predict their appearance using computer graphics techniques. We have
05200 made software which converts from the spine representation to the
05300 polyhedral representation used in our graphics programs to predict
05400 the images of curves for aligning visually. We are taking advantage
05500 of the fact that in many situations, the actions are repetitive (eg
05600 assembly) or that we already know a great deal about the scene by the
05700 time we are manipulating objects, and can use that knowledge to solve
05800 problems which would otherwise be very difficult. The knowledge of
05900 the scene also enables us to work with more reasonable speed, since
06000 we concentrate on small windows in the image. Calibration of visual
06100 instruments is valuable in enabling better prediction of windows.
06200
06300
06400 We are applying this work to visual feedback for assembly. Our basic
06500 approach to visual feedback is to take advantage of the fact that we
06600 already know a great deal about a scene in which we are doing
06700 assembly. We can use that knowledge to predict the appearance of
06800 relevant parts for aligning surfaces, in scenes which are more
06900 complicated than could be understood by our programs without the
07000 knowledge. We are building a visual feedback ability which depends
07100 on a) representing objects within the representation just described,
07200 b) predicting desired curves by converting the object representation
07300 to the usual polyhedral representation as input to a computer
07400 graphics package, and c) matching the predicted curve with edge
07500 points in the image. The first two steps are completed, the final
07600 step is in progress. Visual feedback will thus be model-driven, and
07700 by specifying the mating surfaces,we will be able to generate a
07800 program which aligns the surfaces as desired. This degree of
07900 automating of visual feedback is of great value in programming robots
08000 for assembly. Programming thus becomes a process of describing the
08100 parts to be mated to an interactive system. Of particular interest is
08200 the case of repetitive operations, for which the part description,
08300 transformation of representation and graphics can be done on a
08400 central time-sharing system, while the matching with edge data can be
08500 done on a mincomputer in the assembly module.
08600
08700 Several previous experiments have laid the groundwork for our present
08800 attack on the problem. Wichman [] executed the first robotics visual
08900 feedback system. It was deficient in several ways: only the outer
09000 edges of the blocks were observed; thus the hand had to be removed
09100 from view, and the technique was limited in its domain of objects.
09200 Gill then improved these aspects by recognizing hand marks and
09300 internal edges of blocks. The system was model driven in that
09400 specifying the projected appearance of a corner in a small window
09500 programmed the system for a new task. But the system was limited to
09600 tasks with polyhedra, eg stacking blocks and inserting blocks into
09700 holes. In addition, a coordination of calibration of hand and eye
09800 coordinate systems was used, by tracking the hand through several
09900 positions. A stereo block alignment module was also programmed [].
10000 Perkins [] made an improved corner finder which incorporated the
10100 Hueckel edge operator and identifcation of straight lines by looking
10200 for clusters in a line parameter space. Because the program uses
10300 models of expected images and can find edges with low contrast, the
10400 visual feedback system works moderately well for blocks. We are
10500 thus pushing forward into feedback with curved objects as described
10600 above.
10700
10800 The work on visual feedback depends on display techniques which have
10900 been developed here and elsewhere. An interactive program GEOMED []
11000 was developed here to allow description of objects by building them
11100 up from a set of geometric primitives. The usually tedious task of
11200 input of descriptions of complex objects is much simplified. For our
11300 purposes, a symbolic line drawing output is necessary; a fast hidden
11400 line elimination program with symbolic line output structure has been
11500 written [].
11600
11700 Working with outdoor scenes introduces new difficulties: surfaces are
11800 textured, line finders etc., have been specialized to scenes of
11900 polyhedra. We have made substantial progress by incorporating new
12000 visual modules such as color region finders with a world model. The
12100 first of these efforts [] included a color region finder aad Fourier
12200 descriptors of texture. Textures were described by directionality,
12300 contrast, element size and spacing. These interpretations from the
12400 Fourier transform are useful but not always valid, and a discussion
12500 of the limitations of Fourier descriptors was included. Depth cues
12600 were obtained from the gradient of texture. The results of color and
12700 texture region growing were shown , and a simulation made of
12800 combining these various modes with a world model. An interesting
12900 conclusion was that three dimensional interpretations and cues were
13000 the most semantic help in scene interpretation.
13100
13200
13300 A second project with outdoor scenes has made considerable progress
13400 []. Yakimovsky has made a region-based system which sequentially
13500 merges regions based on semantics of familiar scenes, using
13600 two-dimensional image properties. The system has an intial stage
13700 which is very much like other region approaches, merging regions
13800 based on similarity of color and other crude descriptors except that
13900 it eliminates weakest boundaries first. The second stage introduces a
14000 world model and a means of estimating the best interpretation of the
14100 scene in terms of that model. Interpretation is evaluated by a
14200 "first-order statistical" semantics. An overall linear evaluation
14300 function is made; each region is evaluated not only for its
14400 consistency with an assumed label from the model, but to include
14500 "first-order" consistency of interpretation, according to its
14600 boundary properties with neighbors. Upper and lower limits of
14700 interpretation consistency are evaluated, and a tree formed of merge
14800 choices. The tree is eventually backed up to the best
14900 interpretation. The semantic evaluation provides much better regions
15000 than the same system without the semantics. The system is able to
15100 interactively acquire its statistics of occurrence of region and
15200 boundary properties from sample pictures, what might be called a low
15300 level of learning. It has been demonstrated on road scenes, for
15400 which rather good segmentations and labellings have been achieved.An
15500 application to heart angiograms is in progress.
15600
15700 In human and animal peception, stereo and motion perception of depth
15800 are important. Several of these mechanisms have been programmed.
15900 Nevatia evaluated motion parallax of a moving observer for scenes of
16000 rocks. That program tracked by correlation subsequent images in a
16100 series of images with small angle between images. The technique
16200 allows large baselines for triangulation, and thus is suitable for
16300 depth perception at large distances. Recently, Baumgart has used
16400 motion parallax with features obtained from intensity contours.
16500 Another approach to stereo has been carried out on outdoor scenes [].
16600 Several aspects of stereo have been considered: a) stereo
16700 correspondence in images without calibration. b) establishing
16800 calibration from stereo correspondences, to enable self-calibration
16900 of remotely operated systems; c) stereo correspondence in images with
17000 a well-established relative calibration. A sorting technique for
17100 color and other features saved considerable computation. With
17200 current hardware, brute force techniques are too costly in computing
17300 time for practical utility. Effort has been to choose techniques
17400 which gave considerable gain over brute force techniques and to
17500 evaluate techniques which might become useful with foreseeable
17600 increases in computing power.
17700
17800 A considerable effort has gone into the complex of problems which are
17900 raised in seeking a system which is able to do not just a single
18000 task, but makes it simple to perform many tasks. We have in mind a
18100 vision system for different tasks or environments, or a programming
18200 system to program assembly robots. These problems have led us into
18300 work on strategies, representation, and system design. The system
18400 design has been concerned with introducing and incorporating modern
18500 language features, and communication among cooperating processes, in
18600 spite of limitations of the current time-sharing system. The system
18700 is organized about co-operating modules programmed more or less
18800 independently. Modules have agreed upong communication through a
18900 global data base and message handler. Considerable standardization
19000 has been achieved, so that several users easily work with arm and eye
19100 modules. Recently, we have begun work on systematizing the
19200 bookkeeping of combining vision programs into a system by defining a
19300 set of special vision language primitives within our higher level
19400 language SAIL. These are ways of simplifying the human interface,
19500 but also simplifying the interface with strategy programs.
19600 Eventually, they will lead to an APT-like language for assembly
19700 automation.
19800
19900 We have developed various modules for that system. An edge operator
20000 [] gives a good indication based on local brighness evidence of
20100 whether or not there is an edge near the center of a neighborhood. If
20200 there is an edge, its position and location are relatively well
20300 determined. Differencing techniques have been used to find changes
20400 in scenes. Color region growing techniques have been used.
20500 Accomodation of camera parameters to optimize image quality for each
20600 task has been extensively used. An edge verifier has been used which
20700 is intended to verify or reject hypotheses from an edge proposal
20800 module from one of the body interpretation programs. A focussing
20900 module allows automatic focussing. It could potentially be used in
21000 depth measurement, but its accuracy is much worse than other depth
21100 measurement techniques. Color identification modules have been
21200 written; the latest incorporates the Retinex model of Land to achieve
21300 a certain color constancy, independent of the variation of
21400 illuminance spectrum from scene to scene (sunlight, incandescent,
21500 xenon arc, flourescent) or within a scene (reflections from colored
21600 objects). Several corner finders {Gill,Perkins] have been programmed
21700 for use in visual feedback with blocks. Calibration modules allow
21800 maintaining a system in a well-calibrated state to maintain
21900 reliability, and increase reliability by self-calibration.
22000
22100 Several programs have been written to understand scenes of bloks
22200 given line drawings. An early version [] recognized only outlines of
22300 isolated objects. A later program, COMPLEX, arrived at segmentations
22400 into objects by techniques which were extension of Guzman's but using
22500 lines rather than regions. THis allowed more effectively dealing
22600 with missing and extraneous edges. The system used very limited
22700 prototypes of objects, using size in an important and restrictive way
22800 to identify objects and their spatial relations. Missing edges were
22900 hypothesized for a line verifier program to confirm or reject. Still
23000 another model-based program used moels o a parallelipiped an a wedge
23100 to perform the segmentation into objects corresponding to models. It
23200 is able to handle a variety of missing and extraneous edges. The
23300 thrust of all these efforts is the use of models for perception.
23400
23500